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ABSTRACT 



Decision Support Systems have been evolving for over a decade to assist an 
individual or organization in the decision making process. DSS are often difficult to 
justify or evaluate because many of the benefits that they will provide are intangible 
benefits and are difficult to measure. Over the past decade, there have been several 
methods published that can assist in evaluation or justification. 

In this thesis, I propose a framework designed to assist the user in selecting 
the best available method for evaluation or justification for his/her organization. 
The framework considers: 

• The goals and objectives of the DSS and the organization 

• The tangibility of the goals and benefits 

• The need or requirement to quantify goals and benefits 

DSS are continually evolving and are gaining importance in the information 
systems area. Evaluation and justification will become increasingly more impor- 
tant. The framework will provide a basis for selecting the most appropriate method 
available for an organization. 
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I. INTRODUCTION 



A successful DSS can provide an organization the needed advantage when 
trying to compete in an environment that is financially burdened. The problem 
with DSS is: “How does an organization justify spending dollars on a DSS?” 

To justify a DSS, several methods have been presented throughout literature 
over the past decade. I have designed a framework that will assist the user in 
deciding what method is most appropriate for the user’s organization. 

This framework is designed to take the user through a process of three steps 
or questions that pertain to the organization’s objectives and the purpose of the 
proposed DSS. By following the framework, an organization can select the most 
appropriate method available for not only justifying a DSS, but also for evaluat- 
ing a current DSS. This framework is flexible and can expand to incorporate new 
justification or evaluation methods if and when they become available. 

A. WHAT IS A DECISION SUPPORT SYSTEM 

Decision Support Systems (DSS) are defined as: “A class of information sys- 
tems that draw on transaction processing systems and interacts with other parts of 
the overall information system to support the decision making activities of managers 
and other knowledge workers in organizations.” [Sprague and Carlson, 1982 ] Other 
definitions differ between what an actual DSS is and what the composition of a DSS 
is. Some definitions are very restrictive to the criteria that involve composition of a 
DSS, while others are of a broader scope. 

Restrictive: “DSS are an interactive computer-based system that help decision 
makers utihze data and models to solve unstructured problems.” [Sprague and 
Carlson, 1982 ] 
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Broad: “DSS are any systems that make some contribution to decision mak- 
ing.” [Sprague and Carlson, 1982] 

Both of these types of definitions have drawbacks. The restrictive definition 
greatly limits the DSS category, whereas the broad definition includes any avail- 
able information system. A viable DSS actually falls somewhere between these two 
definitions. 

Decision Support Systems (DSS) have been evolving for the past decade. The 
purpose of DSS is to provide a decision maker with the resources needed for assis- 
tance in the decision-making process. The information obtained from the DSS lends 
credibility to the final decision, but does not become the final decision. The final 
decision involves the DSS data, as well as other factors, and is arrived at by the 
decision maker. One strength of a DSS is that it can focus on semi-structured and 
unstructured problems that are not normally addressed by traditional information 
systems. 

These points are supported by Hogue and Watson [1983], as listed in their 
criteria for a DSS as: 

• Supports but does not replace decision making. 

• Directed toward unstructured and or semi-structured decision 

making tasks. 

• Data and models orgzmized around the decision(s). 

• Easy to use software interface. 

• Interactive processing. 
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TABLE 1.1: DECISION TABLE 



TYPE OF DECISION 


PERCENT OF RESPONDENTS WHOSE 
PRIMARY USE OF THE DSS IS FOR 
THIS TYPE OF DECISION 


General Long Range Planning 


38% 


Strategic Assessment 


23% 


Product Strategy 


8% 


Negotiations of budgets 


8% 


Reporting and Analysis 


8% 


Operational Planning and Control 


6% 


Acquisition Strategy 


2% 


Capital Investment Strategy 


2% 


Financial Strategy 


2% 


General Budgeting 


2% 


Cash Flow Management 


2% 



• DSS use and control is determined by the user. 

• Flexible and adaptable to changes in the environment and the decision maker’s 
style. 

• Quick ad hoc DSS building capabilities. 

B. DSS DECISIONS 

A DSS can assist the decision maker with many types of decisions depending 
on the empheisis of the specific DSS. In a survey, [Meador, et ah, 1984], a DSS was 
primarily used to support the following types of decisions (See Table 1.1): 

C. DSS BENEFITS 

Numerous studies addressed the benefits that an effective DSS could provide. 
[Hogue, et ah, 1985; Keen, 1981; Meador, et ah, 1984; Money, et aJ., 1988). In one 
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survey response, eight top level executives (with little or no DSS experience) listed 
benefits that they perceived to be the most useful: [Alavi, 1982] 

DSS BENEFITS IN DESCENDING ORDER OF USEFULNESS 

• Provide information processing and retrieval capabilities 

• Evaluate alternatives 

• Assist in identifying problems 

• Assist in interpreting information 

• Provide real time analysis of current problems and opportunities 

• Suggest decision alternatives 

• Provide ability to ask “what if” questions 

• Manage executive time by scheduling daily activities 

• Increase decision confidence 

This list does not include all of the benefits that can be derived from an effective 
DSS. However, it is importtint to understand what information top level management 
would like to see when justifying a DSS. 

D. MOTIVATING FACTORS 

What motivates orgtinizations to procure a DSS to assist in the decision making 
process? Benefits are listed in many articles that advocate DSS, but is this enough to 
justify the investment? [Hogue and Watson, 1983] surveyed 18 major corporations 
and listed the motivating factors which led to the development of DSS. 
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• Accurate Information was cited by 12 corporations. 

• Organizational Champion was cited by 8 corporations. 

• New Information was cited by 6 corporations. 

• Managerial Mandate was cited by 4 corporations. 

• Timely Information was cited by 3 corporations. 

• Cost Reduction was cited by 1 corporation. 

While costs and tangible benefits are crucial to an organization, these findings 
on motivation show how important intangible benefits have become in the DSS 
arena. 

E. WHY DO ORGANIZATIONS NEED DSS? 

DSS can provide numerous benefits ranging from increased decision alterna- 
tives to improved communication. DSS that provide increased decision alternatives 
could save (or earn) thousands or even millions of dollars simply by providing the 
decision maker with possible alternatives which may have been overlooked. This is 
just one example of a potential benefit. DSS can also provide sensitivity analysis 
functions to assist a decision maker in choosing between alternatives. The list of 
benefits is long and the value of a successful DSS could be priceless. 

F. WHY DO ORGANIZATIONS NEED TO EVALUATE THEIR DSS? 

While several DSS have become successful, others have been failures. DSS 
are only as good as their input. While it is easy to pick up a magazine and read 
about a successful DSS, it is not as easy to find articles on DSS that were failures. 
Some organizations invest in DSS which ultimately fail. Thus, there must be some 
method to evaluate the success of an organization’s DSS. 
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Evaluation and feedback are very important to any operation. Before an or- 
ganization implements a DSS, an evaluation process should be in place. This can 
help determine if the outcome is worth the investment. Evaluations should be done 
on a continuing basis and then the combinations of the evaluations could be used 
to help justify upgrade a DSS when the time comes to do so. 

G. SCENARIO 

The Commanding Officer for the US Army Recruiting Command knows 
that a decision support system can help decision making tasks such as: 

• From where are the majority of recruits for the army coming? 

• What can be done to increase recruiting in each geographic region? 

• What effect will increased advertising have in certain regions? 

• How much should be spent on advertising? 

The Commander would like to procure a DSS but must justify the expenditure 
to his supervisors. Goals for the DSS are to increase recruitment and to increase 
the overall quality of the individuals that the U.S. Army recruits. 

Analysis to determine proper method of justification for this DSS will be pre- 
sented in Chapter IV. 

H. SUMMARY 

Many articles have been written about the methods used to justify or evaluate 
decision support systems. 

Hogue ^lnd Watson’s survey [1983] of 18 corporations also lists the methods 
used by the participants to justify building the DSS (See Table 1.2): 

These and other justification methods will be examined in Chapter II. 
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TABLE 1.2: JUSTIFICATION OF DSS 



Justification Method 

Intuition 

Value Analysis 

Cost and Hard Benefits 

Cost and Intuitive Benefits 



Percent of Corporations 

44 % 

33 % 

17 % 

6 % 



A notation in the Hogue and Watson survey states, “Ninety-Four percent of 
the corporations surveyed made no formal attempt to measure the financial impact 
of the DSS after it became operational.” 

The purpose of this thesis is to survey the existing methods for DSS justifi- 
cation and evaluation and to present a framework to assist the decision maker to 
decide the most appropriate method available to achieve the corporate needs. 
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II. JUSTIFICATION 



A. INTRODUCTION 

The purpose of this chapter is to provide information on justification methods 
that are available. Justification allows for the prediction of future requirements, 
benefits and costs to evaluate the procurement of a DSS. Several methods of justifi- 
cation of DSS have been published in the literature. The discussion of each method 
is not intended to be a rewrite of the original author’s work. Instead, its purpose 
is to give a user some background information on each method and to provide the 
user with the original author’s work for further elaboration. 

B. JUSTIFICATION METHODS 

The following methods are described in the DSS literature: 

• Excess Tangible Costs 

• Value Analysis 

• Benefit Profile Chart 

• Incremental Analysis 

• Aggregate Benefit Value 

• Profit Index 

• Savings/Investment Ratio 

• Intuition 

• Cost Benefit Ratio 
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1. Excess Tangible Cost 

The Excess Tangible Costs method [Litecky, 1981] is used to quantify 
intangible benefits. The technique itselfis simple, but there are assumptions which 
should be met before using it: 

1. Tangible costs and benefits are relatively easy to estimate. 

2. Intangible benefits are most difficult to estimate. 

3. Tangible costs are ordinarily much greater than tangible benefits. 

4. Intangible costs are insignificant. 

The steps involved in this method are: 

• Step 1. Summarize all of the tangible costs (i.e., program development, in- 
vestment in equipment, etc.). 

• Step 2. Analyze the sum of all tangible benefits (i.e.; cost displacement bene- 
fits, decreased operating costs, etc.). 

• Step 3. Subtract the tangible benefits from the tangible costs. 

This, according to the assumptions, will result in “Excess Tangible Costs”. 
The Excess Tangible Costs is an important figure to management because it 
is the amount which must be overcome by the intangible benefits. 

• Step 4- List the intangible benefits. 

• Step 5. Derive a discount factor for each intangible benefit. The discount 
factor should reflect the probability that the particular benefit will result in 
overcoming one tangible dollar. 
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• Step 6. Multiply the discount factor (probability) by the Estimated Monetary 
Benefit (EMB). This EMB is the amount the benefit would achieve if there 
was 100% probability of achieving success. 

The EMB and the discount factors are derived via a feasibility study. 

Example 



Tangible Costs 
-Tangible Benefits 




($25,000) 

5,000 


Excess Tangible Costs 




($20,000) 


Intangible Benefits 






Benefit #1(10,000) x Discount Factor( .3) 
Benefit #2(10,000) x DF (.5) 

Benefit #3(20,000) x DF (.4) 

Benefit #4(25,000) x DF (.3) 


$3,000 

$5,000 

$8,000 

$7,500 




Total Intangible Benefits 




$23,500 


NET 




$3,500 



Therefore, a combination of tangible and intangible benefits will offset the tangible 
costs by a net amount of $3,500. 

The term Excess Tajigible Costs relates to the dollar amount that must 
be overcome by the intangible benefits. 

2. Value Analysis 

The Value Analysis method was first introduced by Keen [1981] and was 
divided into eleven steps by Smith [1983]. The eleven steps are; 



10 



• Step 1. Define the benefits to be obtained if a prototype system is developed. 

• Step 2. Determine the amount users are willing to pay to obtain the benefits. 

• Step 3. Determine whether a prototype can be implemented within the “cost 
threshold” established by the user. 

• Step 4- Design the prototype and measure its use and cost. 

• Step 5. Review and extend benefits list if necessary. 

• Step 6. Define computer hardware and software which would permit a more 
complete system to be evolved from the prototype. 

• Step 7. Determine cost of expanding the system. 

• Step 8. Ask users to determine level of benefits which must be obtained to 
justify investing in an expanded system. 

• Step 9. Assuming users provide a feasible new “cost threshold”, design a 
second version of the system. 

• Step 10. After the second version is implemented, measure its use and costs 
and determine new cost threshold for a possible third stage of development. 

• Step 11. Continue steps 5-10 until users and systems designers are satisfied 
that the best solution has been achieved under existing constraints. 

This method allows the users to determine what benefits are important 
and place a monetary figure to each benefit. The monetary values are based on 
what the user be willing to pay to obtain a specific benefit? This approach uses 
the “quick hit” approach when developing a DSS and utilizes prototypes to get a 
working model in the users’ hands as quickly as possible and at a low initial cost. 
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3. Incremental Analysis 

Incremental Analysis [Smith, 1983] methodology forecasts the probable 
need of a DSS or Decision Tool, and then determines the various ways to accomplish 
these items without building or buying the DSS. “The first step in using incremental 
analysis is to predict future workloads and to assign probabilities to the forecasted 
workload. Systems personnel then determine alternative methods for accomplishing 
the workload and assign costs to each alternative.” [Smith, 1983] 

Example 

A marketing firm believes there is a 70% chance that they will need a DSS 
in the next two years to eissist in determining a marketing strategy for geographic 
regions and advertising dollars needed to promote the strategy. 

The alternative is that there is a 30% chance that a DSS will not be 

needed. 

Problem: Determine the alternative ways to accomplish the same goal and deter- 
mine costs for each. 

1. Hire additional personnel to determine marketing strategy. 

2. Subcontract to an outside bureau. 





DSS Needed 
(•7) 


DSS Not Needed 
(.3) 


Total Expected 
Cost 


Hire Marketing 


$150,000/year 


0 


$105,000 


Personnel 








Using Outside 


$100,000/year 


0 


$ 70,000 


Consultants 
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From this analysis, the least costly alternative is to hire an outside con- 
sultant at $70,000 per year. If an appropriate DSS be acquired for less than $70,000 
per year, the firm should explore either buying or building a DSS. 

4. Profitability Index 

The Profitability Index method is simply an accounting exercise used to 
determine a project’s worth. The profitability index is a ratio that accounts for cash 
inflows into a project and the amount of investment required. [Garrison 1988] It 
takes into account only the tangible costs and benefits that occur with a project. In 
most cases, DSS provide intangible benefits. This method would be inefficient if an 
organization is interested in any intangible benefits. 

The formula for profitablity index is: 

Present Value of Caish Inflows 
Investment Required 

The higher the profitability index, the more desirable the project. The 
profitablity index it allows for more than one project to be evaluated against other 
projects that may or may not provide the same service. 

Project A 



Hire Marketing Personnel 
Required 



Profitability Index = 1.67 
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Project B 



Purchase of a Marketing DSS 
Required 



Profitability Index = 2.14 



Using this simple example shows how Project B hais the higher Profitabil- 
ity Index, thus, it should be selected over Project A. 

5. Benefit Cost Ratio 

The Benefit Cost Ratio [Naval Data Automation Command (NDAC), 
1980] can be used when costs and benefits are unequal. To determine the economic 
desirability of an investment, the benefits are divided by the costs, providing a single 
figure for comparison. The value represents the amount of benefits obtained per unit 
of cost. 

p/^t> Quantifiable Output Measiire 

Uniform Annual Coat 

• (UAC) Uniform Annual Cost = 

• (QOM) Quantifiable Output Measures = Tangible Benefit 
[Economic Analysis Procedures for ADP, NDAC, 1980] 

The terms and b\f axe factors taken from Appendix B. They are determined by 
the project life. 

Example 

Investment decisions occur randomly over the course of the year. The 
present manual system for evaluating an investment decision takes approximately 3 
days (24 man-hours). An Investment DSS can analyze investment decisions more 
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rapidly, allowing a manager to complete the process in one day (8 man-hours). 
Over the year many investment decisions are overlooked, due to the inability of the 
investment department to keep pace. 

QOM manual = 121 Investment Analysis Opportunities per year 
QOM DSS = 365 Opportunities per year 



Cost Category 


Manual 


DSS 


One-time (year one) 


0 


$50,000 (DSS) 


Recurring (year 2-9) 


$50,000 (salary) 


$50,000 (salary) 



UAC manual = 



50,000(6.042 - .954) 
6.042 - .954 



254,400 

5.088 



$50,000 



UAC DSS = 



50, 000(.954) -h 50, 000(6.042 - .954) 
6.042 - .954 



302, 100 
5.088 



$59,375 



BCR manual = 



QOM 121 
UAC 50,000 



= .00242 



BCR DSS = 



QOM 

UAC 



365 

59,375 



= .006147 



The proposed DSS hcis a higher BCR than the current manual system. 
Therefore, it is the more cost effective alternative. 
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6. Aggregate Benefit Value 

The Aggregate Benefit Value (ABV) method [NDAC, 1980) is one of many 

weighted methods that are available. Weighted methods may be more applicable in 

the DSS arena due to the fact that many of the benefits are intangible. ABV works 

similarly to the Cost Benefit Ratio, method discussed above. 

The example for the ABV method uses the same UAC figures derived 

from the previous example, with the major difference that no QOM can be derived. 
Example: 

The vice-president wants you to evaluate a DSS and compare it to your 
present manual system. His major concerns are the need for a faster response 
time to unexpected investment opportunities and the ability to answer “what if” 
scenarios. Other, less important factors include better use of resources and more 
effective teamwork. (See Tables 2.1 and 2.2) 

Using this method, the DSS again appears more efficient because it has a higher 

BCR. 

The factor weights and the rankings are determined prior to the evaluation. The 
factor weights are determined by the user depending on the value the user hais placed 
on each decision factor. The rankings are then determined by system personnel, 
normally on a scale of 1 to 10. (Faster response time is three times higher for the 
DSS as compare to the manuai method, thus the rankings of 9 for the DSS and only 
3 for the manual method.) 

The Aggregate Benefit Ratio allows a user to factor in the intangible 
benefits, although they <tre difficult to quantify. However, caution must be exercised 
to prevent the introduction of personal opinions and biases. There always exists the 
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TABLE 2.1: BENEFIT RANKING DSS 



Decision Factor 


* Factor Weight 


^Ranking 


Product 


Faster Response Time 


3 


9 


27 


Ability for “What If” scenarios 


3 


7 


21 


Better Use of Resources 


2 


8 


14 


Teamwork 


2 


6 


12 


TOTAL 






74 


UAC (59,375 as determined in CBR example) 






5.9375 


BCR 






12.46 



TABLE 2.2: BENEFIT RANKING MANUAL 



Decision Factor 


* Factor Weight 


^Ranking 


Product 


Faster Response Time 


3 


3 


9 


Ability for “What If” scenarios 


3 


5 


15 


Better Use of Resources 


2 


5 


10 


Teamwork 


2 


5 


10 


TOTAL 






44 


UAC (50,000 as determined in CBR example) 






5.0000 


BCR 






8.8 
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possibility of adjusting factor weight or rankings <is needed. Adjustments will bring 
about a different conclusion. 

7. Savings/Investment Ratio - (SIR) 

A Savings/Investment Ratio (SIR) [NDAC, 1980] is the relationship be- 
tween future cost savings and the investment necessary to obtain those savings. A 
SIR of 1 indicates that the present value of the savings is equal to the present value 
of the investment. For an investment to be economically sound, the SIR value must 
be greater than one. SIR is a characteristic of cost only and does not consider 
benefits (except cost savings). 

SIR is used when evaluating an alternative against the present situation. 



Example; 

Presently you are the head of an investment decision department that 
consists of four employees and four personal computers. The department is respon- 
sible for analyzing investment opportunities for a local bank. The four employees are 
paid $40,000 each per year as a salary. The bank has the opportunity to purchase 
an Investment DSS for $50,000. The Investment DSS would decrease the workload 
of your department by 25%, allowing you to reassign one of your four employees to 
a different department and saving $40,000 per year in salaries that you would have 
had to pay to hire another investment analyst. The DSS has a two year service 
guarantee but possibly could be obsolete after 2 years. 



*SIR = 



PVs 

PVi 



S40, 000 $72,840 

$50,000^ ’ $50,000 



1.4568 



Since the SIR is greater than 1.0, the investment is economically sound. 
The present value of the DSS is greater than the present value of the cost. 
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PVS is the present value of the savings. (All future savings must be 
discounted by using factors from Appendix B to bring the value to a present day 
figure.) 

PVI is the present value of the investment. (If the investment is a one time 
investment that is paid in full there is no need to discount the amount. However, if 
the investment occurs over a specified period of time the the factors from Appendix 
B should be applied.) 

* Project Year Discount Factors such as 1.821 can be found in Appendix B. 

8. Benefit Profile Chart 

The Benefit Profile Chart method [Smith, 1983] uses a comprehensive 
checklist of benefits (See Appendix A) used by system professionals to help in 
preparing justification documents. The checklist can be adapted to any organi- 
zation’s needs. By graphically displaying the benefit charts, a system professional 
can better justify a DSS package to top executives. An example of a Benefit Profile 
Chart can be found in Appendix A. 

9. Intuition 

One of the most-used strategies is simply the use of intuition. [Hogue 
and Watson, 1983] Watson found through an empirical study that 44% of the re- 
spondents to his questionnaire used intuition over any other method when justifying 
procurement of a DSS. 

Also cilong these lines is the motivation to procure a DSS because the 
competition has employed one. Many organizations are motivated by successful 
decision support systems that have been employed by like organizations. The greater 
degree of success from an organization’s competitors, the larger the motivation factor 
for procuring a DSS. In Hogue and Watson’s survey [1983], organizational champion 
was cited by 44% of the corporations as a motivating factor for procuring a DSS. 
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III. EVALUATION 



A. INTRODUCTION 

A current operating DSS should be evaluated with respect to its effectiveness, 
costs, and benefits. This chapter provides information on the evaluation methods 
that are available. The discussion of each method is not intended to be a rewrite of 
the original author’s work. Its purpose is to give a user some background information 
on the method and reference to the original author’s work for further investigation. 

B. EVALUATION METHODS 

The following methods are described in the DSS Literature: 

• Return on Investment (ROI) 

• Expected Value 

• Value Analysis 

• ROI/w Value Analysis or User Satisfaction 

• Profit 

• User Satisfaction 

1. Return on Investment 

The Savings/Investment Ratio [Garrison, 1988] can be used to calculate 
the actual return on investment. The savings/ investment ratio is the relationship 
between cost savings and the initial investment with any operating/maintenance 
costs, to obtain those savings. A SIR of 1 indicates that the present value of the 
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savings is equal to the present value of the investment. For an investment to be 
economically sound, the SIR must be greater than 1. The SIR ratio is a characteristic 
of cost only and does not consider benefits (except cost savings). 

SIR is used when evaluating a new system against the current one. 

Example: 

The investment decision department invested in a DSS three years ago. 
The chief of the department would like to determine if the investment was profitable. 
The DSS the department utilizes had allowed them to eliminate one of the paid 
analyst postions. The cost savings from saved salaries is $40,000 for the first year 
and 42,000 and 43,000 in the second and third years. In the fourth year the DSS 
became obsolete because of several changes in the surrounding environment. The 
department was then forced to hire another analyst. The DSS had provided several 
intangible benefits, including increased alternative choices and faster response time. 
The initial investment was $50,000 with maintenance and upgrade costs of $15,000 
over the four years. 



SIR= ^ 1.92 

Investment $65,000 

Since the SIR is greater than 1.0, the DSS was a sound investment. 

This DSS returned nearly twice the investment. This is using only tan- 
gible costs. The fact that there were also associated intangible benefits makes it 
even more successful. The fact that the DSS was a sound investment although 
it eventually becaune obsolete, could be sufficient justification for investigating the 
procurement of a new DSS. 

Return on Investment could also be cailculated using the formula: [Polokoff, 



1990] 



ROI = 



Benefits — Depreciation 
Costs 
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The DSS in the above example had a 4-year expected life (for deprecia- 
tion purposes), but in the fourth year, the department hired another analyst, thus 
providing no cost savings for the fourth year. The the figures would show: 

Benefits = *12^ = $31,250 
Depreciation = = $12,500 

Costs = $65,000 (50,000 initial -I- 15,000 maintenance) 

ROI per year = = .2885X100% = 28.85% 

ROI = 28.85% 

The organization’s standards would dictate if the ROI was an acceptable 

amount. 

Other accounting methods are also available for use in determining prof- 
itability of a project. The major point to keep in mind with these methods is that 
intangible benefits are overlooked. 

2. Profit 

Profit [Garrison, 1988] is an area that c«in be used to evaluate the success 
of a DSS, but only after an extended period of time. The DSS could operate for 
several years before anyone would be able to attribute the change in profit level to 
the success of the DSS. 

Example 1: 

A successful organization (without the aid of a DSS) enjoys a profit of 
one million dollars per year over three years. A DSS is then built to help them pick 
better investment opportunities. The first year profits increase to 1.2 million. The 
second and third year profits increase to 1.3 and 1.4 million, respectively. 

It could be assumed that the organization appears to be turning a bigger 
profit due to the use of the DSS. However, a part of the increase could be due to 
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many other factors which would have brought about the increase without the DSS’s 
involvement. The true test then would be to operate without the use of the DSS 
and to compare profit levels. Most organizations would not attempt this strategy. 
The best way to evaluate a DSS using this method would entail a thorough analysis 
of all the environmental factors that could possibly lead to increasing or decreasing 
profits. 

Example 2: 

A successful organization (without the aid of a DSS) enjoys a profit of 
one million dollars per year over three years. 

They then build a DSS to help them pick better investment opportunities. 
The first year their profits decreaise to .9 million. The second and third year profits 
decrease to .8 and .7 million, respectively. 

The organization now appears to be losing profits due to the DSS. 

This situation is easier to remedy. If the DSS was scrapped or rebuilt, 
several more years would need to pass to check the profit levels to see if an increase 
results. 

However, environmental factors could be causing the decreased profit. In 
this example, it is easier to doubt the effectiveness of the DSS. Whether or not the 
profits increase or decrease, it would take a long time to determine whether the DSS 
has been successful using this method. 

3. Expected Value 

The accurate use of this method [Smith, 1983] requires an experienced 
and knowledgeable person in the field of statistics, economics, systems analysis and 
DSS to follow these steps: 
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• Step 1: List the DSS benefits and assign them to a group of possible cost 
reduction benefits (Group A, 8, C. . .column 1). 

• Step 2: Assign a possible cost reduction range (column 2) and a probability 
of occurrence (column 3) to each group. 

• Step 3: Multiplying the midpoint of the assigned range (column 4) times the 
total cost of the project (assign the product to column 5). 

• Step 4- Then multiply the probability of occurrence times (column 5) to arrive 
at the probable savings that is listed in (column 6). 



Example 



Group 



Benefit 



Group A Increased number of alternatives 
New insight and learning 
Improved communications 
Control 



Group B Make better use of data 

Better understanding of business 

Group C Ability to carry out ad-hoc analysis 

Fast response to unexpected situations 
Better decisions 



The actual cost measured in dollars from this example was $1,000,000. 
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1 


2 


3 


4 


5 


6 


Grp 


Poss. Cost 
Reduction 


Probability 

of 

Occurrence 


Midpoint 


Cost 

Reduction 
C4 X Total Cost 


Probable 
Savings 
C3 X C5 


A 


0-10% 


70% 


5.0% 


$50,000 


$35,000 


B 


10-15% 


50% 


12.5% 


$125,000 


$62,500 


C 


15-25% 


40% 


20.0% 


$200,000 


$80,000 


Total Probable Savings 










$177,500 



Using this method, it is shown that of the $1,000,000 in actual costs, 
$177,500 is the probable savings from the intangible benefits alone. 

4. Value Analysis 

Value Analysis [Keen, 1981] was discussed in Chapter II. However, value 
analysis can also be used to evaluate a DSS. Once a DSS has been up and running 
in an organization, the question may exist whether to update or scrap the DSS. 
Value Analysis allows the evaluator to determine what the user is willing to pay in 
order to keep his present DSS and to determine if there are other benefits that can 
be added to the DSS. 

Steps 8 through 11 allow the organization to effectively evaluate their 
DSS by going back to the users and discussing expansion or scraping the present 
DSS. 

• Step 8 Ask the users to determine level of benefits that must be obtained to 
justify investing in an expanded system. 

• Step 9 Assuming users provide a feasible new “cost threshold”, design a second 
version of the system. 
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• Step 10 After the second version is implemented, meaisure its use and costs 
and determine new cost threshold for a possible third stage of development. 

• Step 11 Repeat steps 5-10 until users and systems designers are satisfied that 
the best solution has been achieved under existing constraints. 

The evolving DSS that Value Analysis provides is an excellent way of 
continually evaluating an organizations DSS. If users are pleased with the initial 
versions of the DSS, the organization can continually update the DSS to include 
more functions. 

5. User Satisfaction 

Several methods are available to measure user satisfaction. [Baroudi and 
Orlikowski, 1988] However, a common method is the use of a questionnaire-type 
survey. Surveys can help the organization determine the features of a DSS which 
are the most beneficial, used, or successful and it can also help determine those 
features that are worthless or dissatisfying. There are cases where DSS tools and 
user satisfaction surveys have not produced consistent results. These studies showed 
that user satisfaction surveys can sometimes be deceiving. In an assessment [Aldag 
and Power, 1986] of computer-assisted decision analysis, users thought favorab: . of 
the decision tool, however using the decision tool did not improve their decisions. In 
a separate study [Gallupe and DeSanctis, 1988], the decision tool greatly enhanced 
the quality of the decision, however, satisfaction and confidence were lower than the 
group that did not use the decision tool. Despite these findings, user satisfaction 
surveys are still widely used and accepted. Most surveys use a Likert-type scale, 
allowing the respondent to evaluate on a scale of 1 to 7, the features of the DSS. Ap- 
pendix C shows an example of a User Satisfaction Survey. [Baroudi and Orlikowski, 
1988] 
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6. ROI w/Value Analysis or User Satisfaction 



This method combines two methods [Baroudi and Orlikowski, 1988] [Gar- 
rison, 1988] and attempts to balance the tangible benefit side (ROI) with the intan- 
gible benefits (Value Analysis or User Satisfaction). 

The methods have been described above. An organization could run 
both methods to determine the bottom line return on their investment and to also 
give consideration to intangibles by implementing value analysis or user satisfaction 
techniques. By implementation of this method, the organization would evaluate 
their present DSS and would also provide data for future justification to updated 
or modify the current DSS. By implementing value analysis or user satisfaction 
techniques, it will be easier to overcome a lower than expected ROI (provided that 
the user satisfaction level was high with the initial DSS). 

This method help to satisfy those who want to see a bottom line figure, 
while at the same time taking into account the intangible benefits that are always 
present in the DSS arena. 
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IV. A FRAMEWORK FOR DSS 
EVALUATION AND JUSTIFICATION 

A. INTRODUCTION 

There are several ways to evaluate or justify an investment. Procurement of 
any system, whether a DSS or a traditional information system, should be thought 
of as an investment. This chapter presents a framework that will assist a user in 
selecting an appropriate method, or set of methods, to evaluate or justify a DSS. 
The framework that I have designed is a multi-level decision tree. (Figure 4.1) 

The decision tree’s purpose is to guide the user through 3 levels of questions 
to arrive at a methodology which will assist his/her organization when evaluating 
or justifying a DSS. The questions within the decision tree are designed to separate 
the available methods. The answers to the questions within the decision tree should 
be based on the user’s organizational goals. Upon answering the questions, the user 
will arrive at a methodology which will best fit organizational needs. 
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Figure 4.1: Framework For Evaluating or Justifying DSS 
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B. LEVEL 1 



The first level of the framework addresses the purpose of the DSS assessment. 
Is the user trying to justify future procurement (buy or develop) a DSS, or is he 
looking to evaluate the current DSS in his organization? Several methods are avail- 
able to justify and to evaluate a DSS, however, most methods apply only to either 
justification or evaluation. 

C. LEVEL 2 

The second level in this framework deals with the objectives to be accomplished 
within the DSS. Are the goals mostly tangible goals (e.g., time savings, cost savings, 
profit) or are they mostly intangibles (e.g., better decisions, more alternatives, user 
satisfaction, improved service for the customer)? 

The key to answering these questions is in knowing the organization’s mission 
or goals. If an organization’s goal is strictly increased profit, this goal is relatively 
easy to quantify. However, while increased profit is the primary goal, many or- 
ganizations have secondary goals. If the organization wants increased profits with 
increased organization morale, the goals are not as easily quantifiable. 

When answering the Level 2 question, the adjective “mostly” should be con- 
sidered. 

If the organization’s primary goals are mostly tangible with some intangible 
secondary goals, then it should take the the “tangible route.” If the organization’s 
primary goals Me mostly intangible with tangible secondary goals, then the “intan- 
gible route” should be followed. 

In the case where the organization’s goals are unknown or the user cannot 
decide between tangible and intangible, the route label “both or unknown” should 
be chosen. However, this route will require the user to do a thorough analysis of 
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many methods to justify the DSS in order to find the method that best fits the 
organization. 

Many DSS are geared toward functional areas and may be far removed from 
the primary goals of an organization. In this case, the functional area goals need to 
be addressed in determining the most appropriate route in the framework. 

D. LEVEL 3 

The third level of the proposed framework leads the user to a method or a 
set of available methods considered to be the most appropriate to the user’s given 
situation. 

The following routes are available for DSS Justification. 

1. Justifying/Tangibles 

If a user is trying to justify purchasing or developing a DSS and the 
benefits are mostly tangible, the third question is “Do intangible benefits have any 
impact on goals?” If the intangible benefits do have some impact, then the “yes” 
route will lead to the following methods of justification: 

• Excess Tangible Cost [Litecky, 1981] 

• Value Analysis [Keen, 1981] 

If the intangible benefits have little or no impact on goals, the “no” route 
will lead to the following methods: 

• Profitability Index [Garrison, 1988] 

• Cost Benefit Ratio [NDAC, 1980] 

• Savings Investment Ratio [NDAC, 1980] 

• Incremental Analysis [Smith, 1983] 
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2. Justifying/Intangibles 

If the user is trying to justify a DSS and the goals are mostly 

intangible, the question to be addressed will be; “Is there a need to 
quantify intangible benefits?” An organization may desire a DSS that provides in- 
formation for better decisions (intangible), however, it wants to show a “bottom line 
figure” (tangible). The following methods are available to help give the organization 
that “bottom line figure”: 

• Excess Tangible Costs [Litecky, 1981] 

• Aggregate Benefit Value (weighted methods) [NDAC, 1980] 

If the organization does not require a bottom line figure, the following 
methods are more appropriate: 

• Value Analysis [Keen, 1981] 

• Benefit Profile Chart [Smith, 1983] 

• Intuition [Hogue and Watson, 1983] 

• Success Stories 

These methods are discussed in Chapter II. 

3. Evaluate/Tangible 

When evaluating a DSS designed for mostly tangible benefits with some 
intangible impatct on goals, Return on Investment in conjunction with User Satis- 
faction or Value Analysis would be appropriate. 

If the intangible benefits do not affect goals, simply Return on Investment 
(Actual) or Profit would be better measures. 
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4. Evaluating/Intangibles 

If an organization is evaluating a DSS designed for intangible benefits and 
there is still a need to quantify the answer, then the Expected Value Method would 
be appropriate. 

If there is not a need to quantify an evaluation, the following methods 
would be more appropriate: 

• User Satisfaction [Baroudi and Orlikowski, 1988] 

• Value Analysis [Keen, 1981] 

E. SUMMARY 

Organizations may have any number of goals. An organization may have a 
goal that is tangible in nature but the user desires a DSS that provides intangible 
benefits (e.g., more alternatives to help produce better decisions). The objective of 
the DSS now falls into the realm of intangibles even though the organization’s goals 
are tangible. To successfully use the decision tree, both the goals of the organization 
and the proposed purpose of the DSS must be taken into account and the appropriate 
measures applied cis listed above. 

What follows are three scenarios which demonstrate the usefulness of the pro- 
posed framework. 
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F. SCENARIO 1 



The Commanding Officer for the U.S. Army Recruiting Command knows that 
a decision support system can help him with several decision making tasks such as; 

• From where are the majority of recruits for the U.S. Army 
coming from? 

• What can be done to increase recruitment in each geographic 
area? 

• What effect will increased advertising have in certain 
regions? 

• How much should be spent on advertising? 

The commander would like to procure a DSS, but must justify the 
expenditure to his superiors. Goals for the DSS are to increase recruitment 
and to increase the overall quality of individuals that the U.S. Army recruits. 

In this scenario the best response would be: 

Level 1. Justification 

The organization would like to procure a DSS. The DSS will require justifica- 
tion. 

Level 2. Tangible 

The scenarios goals are for increased recruitment and increased quality of 
recruits. These goals axe tangible and can be measured. The recruiting command 
administers a aptitude test to judge overall quality of the recruit. These goals can 
be measured using traditionad Cost Benefit Analysis techniques. 
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Level 3. 



The answer to the level three question is “No”, intangibles benefits do not 
impact on goals. 

The scenario does not state any intangible benefits as either primary or sec- 
ondary goals. 

This information leads the analysis to the following applicable alternative jus- 
tification methods: 

• Incremental Analysis [Smith, 1983] 

• Profitability Index [Garrison, 1988] 

• Cost Benefit Ratio [NDAC, 1980] 

• Savings/Investment Ratio [NDAC, 1980] 
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G. SCENARIO 2 



The human resource manager controls pay and suggests appropriate increases 
for each position within the organization. He would like a DSS to assist in the 
decision-making process. The DSS would be a great timesaver, but would primarily 
be used to satisfy the organization’s members and help keep morale at a high level. 

The DSS would have to be able to assist in dividing up the budget dollar for 
annual pay raises on a semi-annual and annual basis. Keeping employees happy is 
an important goal since to alienate them would contribute to the loss of outstanding 
employees. 

The DSS should be able to capture market information on what other organi- 
zations are paying (this can be accomplished through a financial service). 

The DSS should be able to incorporate the consumer price index. It should be 
able to show where the organization stands against the market on current salaries. 

It should be able to answer questions such as ”What if I want to increase all 
programmers’ salaries in my organization to account for the difference in their pay 
and the market leader? How much would this increase costs to the organization?” 

Top management would like the tangible benefits quantified and a list of all of 
the intangible benefits that the DSS will provide, but there is not a need to attempt 
to quantify improved morale and other intangible benefits. 

In this scenario, the best response would be: 

Level 1. Justification 

The organization would like to procure a DSS which requires justification. 

Level 2. Intangible 

The DSS would be used primarily to assist in the decision making process, to 
save valued time and to keep organization’s members motivated. The scenario lists 
several intangible goals but makes no mention of intangible goals. 
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Level 3. 



The answer to the level three question is “No” there is not a need to quantify 
the intangibles benefits). 

Top management would like the tangible benefits quantified, but the intangible 
benefits should be listed. There is no need to place a specific dollar value to the 
intangible benefits. 

This information leads the human resource manager to these applicable alter- 
native justification methods: 

• Value Analysis [Keen, 1981] 

• Benefit Profile Chart [Smith, 1983] 

• Intuition [Hogue and Watson, 1983] 
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H. SCENARIO 3 



A major airline is currently running a decision support system for assisting 
allocation of planes to different flights and scheduling passengers for each flight. 
The DSS cissists the decision-maker in insuring that planes are available for every 
flight. If a plane breaks down, the DSS can find another available plane to meet the 
grounded plane’s commitments. 

The primary goal of the DSS is to increaise the airline’s profits through in- 
creased customer satisfaction. The better the organization’s ability to meet all of 
its commitments, the more satisfied the customers should be. The assumption is 
made that when an airline hcis to cancel flights because of breakdowns, customers 
will begin to switch to a more reliable carrier. 

Management wants to know if an upgrade of the present system is needed. 
While the DSS in place is doing a satisfactory job, there is a question as to whether 
a newer system would be better. 

In this scenario, the best response would be: 

Level 1. Evaluation 

In the scenario, the airline presently has a DSS and would like to evaluate the 
DSS to assist the airline in making decision on whether to invest more money on 
upgrading their system. 

Level 2. Tangible 

The goals in this scenario are centered around increcising the airlines profits. 
Level 3. 

Yes (Intangibles Benefits have an impact on goals) 

The primary goal is to increase profits. However, customer satisfaction does 
have an impact on the goals of increased profits. 
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This information leads the organization to the following analysis 

• ROI w/Value Analysis 

• ROI w/User Satisfaction 
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V. CONCLUSION 



A. THERE IS A NEED FOR DSS 

Decision Support Systems can play a vital link in the area of decision making. 
DSS are not intended to make final decisions, rather to assist the decision maker in 
the decision making process. Expanding a decision makers views and alternatives, 
combined with the ability to run sensitivity analysis on each alternative, can greatly 
enhance the quality of a decision. However, not every DSS is successful. There must 
be some means set in place to justify and evaluate a DSS. Even if the evaluation 
is nothing more than a user satisfaction survey, this is still a method which will 
provide feedback to top-level management. Without some sort of feedback method, 
the organization will simply run blind. 

In Chapter III, several methods of justification and evaluation 
are presented. Each method has advantages and disadvantages. 

B. COST BENEFIT METHODS 

“Cost/benefit evaluations will probably provide the most meaningful results to 
management.” [Sprague and Carlson, 1982] However, CBA runs into trouble when 
trying to evaluate or justify intangible benefits. 

C. VALUE ANALYSIS 

This method is simple to implement, using prototyping which can help limit 
the initial investment while testing the water to see if the system will be a success. 
However, it is “less rigorous than cost benefit techniques and may not include all 
the measures that are relevant.” [Sprague and Carlson, 1982] 
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D. USER ATTITUDE/SATISFACTION 



Using surveys to measure user satisfaction can also run into problem areas. 
[Sprague and Carlson, 1982] lists the following problem areas when using question- 
naires: 

1. In developing a questionnaire to measure attitudes, one may be trying to 
quantify that which is not quantifiable and the subsequent analysis may be 
misleading. 

2. Questionnaire respondents may interpret questions differently than intended, 
and answers to some questions may influence others. 

3. Administering questionnaires may be an inconvenience to individuals and ex- 
pensive (in terms of hours lost by the respondents and hours spent by the 
interviewers). 

4. The method does not identify the causes of any measured changes. 

E. OTHER METHODS 

• Expected Value 

• Incremental Analysis 

• Excess Tangible Costs 

• Aggregate Benefit Value 

Depending on the organization and its goals, these methods can be very ben- 
eficial and simple to use. However, they tend to rely on subjective judgement when 
establishing weighting or probability of occurrence figures. In some ca.ses, the slight- 
est change in weights or probabilities can drastically change the outcome of the 
evaluation. 
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F. USE OF THE DECISION TREE 

The main purpose of this thesis was to provide a framework that can be used to 
assist the decision maker in determining the most appropriate method for evaluating 
or justifying a DSS. The decision tree that is provided in Chapter IV (along with a 
description of the methods) allows a decision maker to quickly assess the methods 
that best suits his personal or organizational needs. The methods for evaluating 
and justifying a DSS will continually change. The decision tree can be incorporated 
to handle any changes. The basic premise for the decision tree is based on three 
questions that the organization must answer: 

1. Is the decision maker attempting to justify future procurement of a DSS or is 
the decision maker attempting to evaluate the current DSS? 

2. Are the goals of the organization mostly tangible or intangible? If the DSS is a 
personal DSS for an individual, then the individual’s goals should be analyzed. 
Hopefully, the individual decision makers goals complement the organizational 
goals. 

3. If the goals are mostly tangible, do the intangible benefits have any impact on 
obtaining the organizational goals? 

4. If the goals are mostly intangible, is there a need to quantify intangible bene- 
fits? 

By successfully answering these questions, the decision tree should lead the 
decision maker to a method(s) that will provide the best available results given the 
organization’s views and goals and prove acceptable to top-level management. 
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G. SIX POINTS 



While attempting to justify or evaluate a DSS, the following points should be 
kept in mind: 

1. DSS can provide both tangible and intangible benefits. All appropriate bene- 
fits should be considered when the justification process begins. 

2. Benefits should be viewed in light of the organizational objectives. 

3. The use of a personal DSS should be viewed in light of the user’s objectives, 
and should coincide with that of the organization. 

4. Evaluation of a DSS is an important feedback mechanism that helps to provide 
information to top-level management. 

5. Evaluations throughout the lifecycle of the DSS are necessary to minimize the 
risks of continually pouring money into an unsuccessful system. 

6. Doing one or two things well is far better than doing several things that are 
mediocre. Do not try to provide a DSS that delivers everything because it can 
truly cost everything. 
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APPENDIX A 

Cost Benefit Profile Chart [20] 

Intangible benefits of MIS/DSS leading to improved business performance. 

• Category A = Some improvement in existing system. 

• Category B = Significant improvement in existing system. 

• Category C = Significant new benefit. 
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Business Function/ Activity 


Degree of Performance Improvement 


Engi nee ring/ Research 


A 


B 


C 


Interactive Problem Solving within 
Company 








Interactive Problem Solving with 
Customer 








Stimulation of new ide«is 
(i.e., graphics) 








Faster Design (i.e., computer-aided 
design) 








Control of Specifications/drawing 








Access to technical information 








Processing of engineering change 
orders 








Manpower/Project Management 








Management of Professional’s time 
e.g., reduced clerical workload 
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Business Function/Activity 


Degree of Performance Improvement 


Finance/ Accounting 


A 


B 


C 


Budget Preparation 








Use of operating/leverage 








Privacy of data information 








Security of data information 








Integrity (accurax:y of data) 








Planning/ Control of liquid assets 








Capital Budgeting 








Auditing and Internal Control 








Simplified Reporting 








Timely Reports 
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Business Function/ Activity 


Degree of Performance Improvement 


Employee Relations/Human Resources 


A 


B 


C 


Identification of best performers 
(Individual and Groups) 








Strategic Manpower Planning 








Places and methods for recruiting 








Improved government reporting 








Monitoring of EEO/ERISA/OSHA 
Standards 








Higher motivation for workforce 








— Career planning capabilities 








— Turnover/absenteeism 








— Fringe benefits planning/control 








— Job satisfaction 








Employee Training 

e.g., computer assisted instructions 
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Business Function/Activity 


Degree of Performance Improvement 


Employee Relations/Human Resources 


A 


B 


C 


Understanding of how HRM functions 








Wage and Salary Planning and Control 
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Business Function/ Activity 


Degree of Performance Improvement 


General Management 


A 


B 


C 


Increased Communications among 
departments 








Planning data more quickly/easily 
assessible 








Ability to provide “what if” reports 








Faster development of new systems 








Easier to use system 








Increased secretarial efficiency, 
effectiveness 








Better Meetings 








Provides better reliability 








Improved Accuracy/Conciseness 
Timeliness relevance of all 
information 








Cost Avoidance 








— Precludes need to hire new people 
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Business Function/Activity 


Degree of Performance Improvement 


General Management 


A 


B 


C 


— Need fewer computer programs 








— Need less program maintenance 








— Reduced travel costs 








— Better use of programmers time 









Business Function/Activity 


Degree of Performance Improvement 


Decision Support System 


A 


B 


C 


— Goal seeking 








— What if stimulation 








— Graphics/modeling 









The Cost Benefit profile is not meant to fit every aspect of a particular organization. It 
must be tailored to a specific organication. This is strictly a guideline of a Cost Benefit 
Profile. 
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APPENDIX B 

Project Year Discount Factors 



TABLE A 



TABLE B 



Present Value of $1 (Single 
Amount: Used when cash flows 
accrue in varying amounts 
each year) 



Present Value of $1 (Cumulative 
Uniform Series: To be used 
when cash flows accrue in the 
same amount each year) 



Project 


10% 


10% 


1 


0.954 


0.954 


2 


0.867 


1.821 


3 


0.788 


2.609 


4 


0.717 


3.326 


5 


0.652 


3.977 


6 


0.592 


4.570 


7 


0.538 


5.108 


8 


0.489 


5.597 


9 


0.445 


6.042 


10 


0.405 


6.447 


11 


0.368 


6.815 


12 


0.334 


7.149 


13 


0.304 


7.453 


14 


0.276 


7.729 


15 


0.251 


7.980 



Note: Table B factors represent the cumulative sum of Table A factors through any 
given project year. 

Reference: [NDAC, 1980] 
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APPENDIX C 

User Satisfaction Survey 

This appendix provides a basic user satisfaction survey [Baroudi and Or- 
likowski, 1988]. The survey is not intended to meet every organizations needs. 
However, it does provide the basis for a user satisfaction survey, which should be 
tailored to an organization’s needs and goals. 

This survey should be administered to the users of the DSS after weights have 
been placed on each of the possible responses. Likert scale normal runs from 1 
to 7 points. It is up to the organization to establish an acceptable score prior to 
administering the survey. 

Please follow these instructions: 

1. Check each scale in the position that describes your evaluation of the factor 
being described. 

2. Check each scale, do not omit any. 

3. Check only one position for each scale. 

4. Check in space, not between spaces. 

ANSWER BASED ON YOUR OWN FEELINGS 
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1. Relationship with the Management Information Department (MID) staff 



dissonant : : : : : ; : : harmonius 



bad : : : : : ; : : good 



2. Processing of requests for changes to existing DSS 



fast : : : : : : : : slow 



untimely : : ; : : : : : timely 



3. Degree of training provided to users 



complete : : : : : ; : : incomplete 



low : : : : : : : ; high 



4. User understanding of the DSS 



insufficient : : : : : : : : insufficient 



complete : : : : : ; : : incomplete 
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5. User’s feeling of participation 



positive : : : : : : ; : negative 



insufficient : t : : : sufficient 



6. Attitude of the Management Information Department staff 



cooperative : ; : : : : : : belligerent 



negative : : : : ; ; : : positive 



7. Reliability of output information 



high : : : : : : : : low 



superior ; : : : : : : : inferior 



8. Relevancy of output information (Does the DSS do intended function) 



useful : : : : : : : : useless 



relevant : : : : : : : : irrelevamt 
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9. Accuracy of output information 



inaccurate : ; : : : : : : accurate 



low : : : : : : : : high 



10. Precision of output information 



low : : : : ; : : : high 



definite : : : : : : : : uncertain 



11. Communication with the Management Information Department staff 



dissonant : : : : : ; ; : harmonious 



destructive : ; : : ; : : : productive 



12. Time required for the new systems development 



\inreasonable : : : ; : : : ; reasonable 



acceptable : : : : : : : ; unacceptable 
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13. Completeness of the output information 



sufficient : : : : : : : : insufficient 

adequate ; ; : : : : : : inadequate 

14. DSS assisted in decision making process 

definite : : : : : : : : uncertain 

superior ; : : ; : ; : : inferior 

15. Is DSS easy to learn and use 

easy : : : ; : : : : difficult 

fast : : : ; : : : : slow 

16. List the functions that are the most beneficial: 

17. List the functions that are most needed: 
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